The Noisy Channel Coding Theorem
نویسنده
چکیده
A self-contained proof is given of the Noisy Channel Coding Theorem as described in Chapter 8 of Elements of Information Theory by Thomas M. Cover and Joy A. Thomas (first edition). 1. Definitions The channel and its capacity. We assume a channel with transition probabilities p(Y |X) for the finite input alphabet X and finite output alphabet Y. For each marginal probability distribution p(X) over X we have a joint distribution p(X,Y ) (defined by p(x = X, y = Y ) = p(x = X)p(y = Y |x = X)) and marginal distribution p(Y ) (defined by p(y = Y ) = ∑ x p(x = X, y = Y )). With this probability distribution comes the entropy H(X) defined the usual way: H(X) := ∑ x∈X −p(x) log p(x). This definition can also be written as the expectation H(X) = Ep[− log p(X)]. Similarly we have H(X,Y ) = ∑ x,y −p(x, y) log p(x, y) = Ep[− log p(X,Y )] and H(Y ) = ∑ y −p(y) log p(y) = Ep[− log p(Y )]. Given a distribution p(X) we can thus also speak of the corresponding mutual information I(X;Y ) := H(X) + H(Y ) − H(X,Y ) = H(Y )−H(Y |X) = H(X)−H(X|Y ) between X and Y . The capacity C of the channel is defined by C := max p(X) I(X;Y ), which is a property of the conditional probability p(Y |X), independent of the marginal distribution p(X). Jointly Typical Sets. The probability of strings x ∈ X of length n is also denoted by p(x) such that p(x) := p(x1) · · · p(xn). For the joint probability distribution p(X,Y ), length n and error ε > 0, the typical set A (n) ε is defined by A ε :=
منابع مشابه
The Noisy-Channel Coding Theorem
This is an exposition of two important theorems of information theory often singularly referred to as The Noisy-Channel Coding Theorem. Given a few assumptions about a channel and a source, the coding theorem demonstrates that information can be communicated over a noisy channel at a non-zero rate approximating the channel’s capacity with an arbitrarily small probability of error. It originally...
متن کاملLecture 3 : Shannon ’ s theorem and some upper and lower bounds
1. Noisy Channel: This channel introduces a noise in the data during transmission and hence the receiver gets a noisy version of the transmitted data. Therefore, for reliable communication, redundancy has to be added to the data generated by the source. This redundancy is then removed at the receiver’s end. This process of introducing redundancy in the data to make it more resilient to noise is...
متن کاملOn Interactive Computation over a Noisy Channel
of the Dissertation On Interactive Computation over a Noisy Channel
متن کاملSecure multiplex coding to attain the channel capacity in wiretap channels
It is known that a message can be transmitted safely against any wiretapper via a noisy channel without a secret key if the coding rate is less than the so-called secrecy capacity CS , which is usually smaller than the channel capacity C. In order to remove the loss C−CS , we propose a multiplex coding scheme with plural independent messages. In this paper, it is shown that the proposed multipl...
متن کامل